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Abstract. We show that the open unit ball of the space of operators from a finite dimen- 
sional Hilbert space into a separable Hilbert space (we call it "operator ball" ) has a restricted 
form of normal structure if we endow it with a hyperbolic metric (which is an analogue of 
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the standard hyperbolic metric on the unit disc in the complex plane). We use this result 
to get a fixed point theorem for groups of biholomorphic automorphisms of the operator 
ball. The fixed point theorem is used to show that a bounded representation in a separable 
Hilbert space which has an invariant indefinite quadratic form with finitely many negative 
squares is unitarizable (equivalent to a unitary representation). We apply this result to find 
dual pairs of invariant subspaces in Pontryagin spaces. In the appendix we present results 
of Itai Shafrir about hyperbolic metrics on the operator ball. 
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1 Introduction 

Let H be Hilbert spaces; by L{K^ H) we denote the Banach space of all linear bounded 
operators from K to H. We will denote the open unit ball of L{K^ H) by B and call it 
operator hall. We say that a subset M of S is separated from the boundary if it is contained 
in a ball ri3, for some r G [0, 1). 

A group G of transformations of B will be called elliptic if all its orbits are separated 
from the boundary (this terminology goes back to [9j). 

We call G equicontinuous if, for each e > there is 5 > such that ii A, B ^ B and 
\\A — B\\ < 6 then ||fi'(A) — g{B)\\ < e for all g E G. This condition can be also called global 
equicontinuity because it is possible also to consider equi continuity in a point. 

Since ;B is a bounded open set of a Banach space, one may consider holomorphic maps 
from B to Banach spaces. We will deal with invertible holomorphic maps from B onto B; 
such maps are called biholomorphic automorphisms of B. Our aim is to prove that if one of 
the spaces K, H is finite-dimensional and the other is separable, then any elliptic group of 
biholomorphic automorphisms of B has a common fixed point. More precisely we will prove 
the following result. 

Theorem 1.1. Let dimi^ < oo and H be separable. For a group G of biholomorphic 
automorphisms of B, the following statements are equivalent: 

(i) G is elliptic on B; 

(ii) at least one orbit of G is separated from the boundary; 

(iii) G is equicontinuous; 

(iv) G has a common fixed point in B. 

Remark 1.2. The assumption dimi^ < oo is essential, some of the results of this paper are 
known to fail without it, see, for example, the last paragraph of Section\^ As for separability 
of H, it is just a technical convenience, our approach works for non-separable H also, with 
a bit more complicated proofs. 
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The result will be applied to the orthogonalization (or similarity) problem for bounded 
group representations on Hilbert spaces. This problem can be formulated as follows. Let 
TT be a representation of a group G on a Hilbert space Ti. Under which conditions there 
is an invertible operator V such that the representation a of G, defined by the formula 
(T{g) = Vn^g)}^'^, is unitary? 

Clearly a necessary condition is the boundedness of vr: sup^^^ ll^lfi')!! < oo- general 
it is not sufficient. Some sufficient conditions (on G or tt) are known, see the book [12]. We 
will show that a bounded representation vr of a group G on a Hilbert space TC is similar to 
a unitary representation if it preserves a quadratic form t] with finite number of negative 
squares. The last condition means that T]{x) = \\Px\\^ - IIQxf and P,Q are orthogonal 
projections in H with P + Q = 1 and dim^QTi.) < oo. 

As a consequence we obtain that each bounded group of J-unitary operators on a Pon- 
tryagin space H^ has an invariant dual pair of subspaces. In other words the space can be 
decomposed into J-orthogonal direct sum H^ + H^ of positive and negative subspaces which 
are invariant for all operators in the group. 

The proof of Theorem 1 1.1 1 is based on the analysis of the structure of the operator ball as 
a metric space with respect to the Caratheodory distance (see Chapters 4 and 5 of [6J). It 
was proved by Shafrir yjj that i3 is a hyperbolic space with respect to this distance. Since 
[U] is not easily accessible, we present a proof of this result in an appendix to our paper, 
with the kind permission of the author. We will show that B has a restricted form of a 
normal structure if dim{K) < oo. 

In the case where K is one- dimensional Theorem 11.11 was obtained in [17] ; a transparent 
proof can be found in [lOl Section 23] . 

2 Hyperbolic spaces 

In our definition of hyperbolic spaces we follow fixed point theory literature (see e.g. [13] , 

[H] ). In geometric literature (see e.g. [3j) hyperbolic spaces are defined differently. 

By a line in a metric space {X,p) we mean a subset of X which is isometric to the real 
line M with its usual metric (in the literature lines are also called metric lines or geodesic 
lines) . 

Let {X,p) be a metric space with a distinguished set of lines. We say that X is a. 
hyperbolic space if the following conditions are satisfied: 

(I) (Uniqueness of a distinguished line through a given pair of points) For each x,y & X, 
there is exactly one line i & A4 containing both x and y. 

(2) (Convexity of the metric) To state the condition (see (12.31) ) we need to introduce some 
more definitions and notation. The segment [x, y] is defined as the part of the line i & A4 
containing both x and y, consisting of all z & £ satisfying 

p{x,y) = p{x,z) + p{z,y). (2.1) 
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We write 

z={l-t)x®ty (2.2) 

ii z e [x,y], p{z,x) = tp{x,y), and p{z,y) = (1 - t)p{x,y) (where t E [0, 1]). 
The convexity condition is: 

Hyperbohc spaces satisfy also the following stronger form of the condition (12.31) : 

p((l - t)x © ty, (1 - t)w © te) < (1 - t)p(x, w) + tp(?/, 2;). (2.4) 

(To get (12. 4p from (12. 3p we observe that, if for some value of t we have the inequalities 
p{{l-t)x®ty, {l-t)x®tz) < tp{y,z) and p((l-t)x©te, {l-t)w®tz) < {l-t)p{x,w), then, 
by the triangle inequality, we have (12. 4p for that value of t. Using this observation repeatedly 
we prove the inequalities from this paragraph for t of the form ^ {k E N,l < k < 2"). Then 
we use continuity.) 

A subset C C A:' is called convex if x,y E C implies [x, y] C C . Sometimes we say 
p- convex instead of convex, to avoid confusion with other natural notions of convexity for 
the same set. We use the notation Ea^r for {x E X : p{a,x) < r} and call such sets closed 
balls. The condition (12.41) implies that in a hyperbolic space all closed balls are convex. 



3 Normal structure 

Let M be a subset in a metric space {X, p). The diameter of M is defined by 

diamM = sup{p(x, y) : x,y E M}. (3.1) 

A point a G M is called diametral if 

sup{p(a, x) : X E M} = diamM. 

A hyperbolic space X is said to have normal structure if every convex bounded subset of 
X with more than one element has a non-diametral point. 

This notion goes back to Brodskii and Milman |4j who proved that uniformly convex 
Banach spaces (which are hyperbolic spaces) have normal structure. Takahashi [18] intro- 
duced and studied normal structure in somewhat more general context. See [2", Chapter 3] 
for a nice account on those aspects of fixed point theory which are related to the geometry 
of Banach spaces. 

Lemma 3.1. Let M be a separable bounded convex subset of a hyperbolic space X and a be 
the diameter of M. If all points of M are diametral then M contains a sequence {an} with 
the property: lim„^oo p(fln, a^) = « for each x E M. 
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Proof. Let {c„} be a dense sequence in M. We define a sequence of "centers of mass" 
by the following rule: bi = Ci, = :^bn © ;^c„+i. By convexity of p we have 

1 " 

p(a;, fen) < - c^) (3.2) 

n 

A;=l 

for all n G N. Indeed for n = 1 this is obvious. If it is true for some n, then p{x,bn+i) < 

By convexity of M we have bn & M for each n G N. Our assumption implies that 
bn is diametral, hence there is a point an & M with p{bn,an) > (1 — :;|2)«- It follows that 
(l-^)a < ^ ELiP(«n,Cfc). If p(a„,Cj) < (l-^)a, for some j < n, then ^ XlLi P(«n> Cfc) < 
^(1 — + = (1 — a contradiction. Hence p(an, Cj) > (1 — for j < n. This 

shows that lim„^oo p(«n, c^) = a for each fixed j. Since the sequence {cj} is dense in M, the 
lemma is proved. □ 



4 The invariant distance in the operator ball 

Recall that K, H denote Hilbert spaces and B is the open unit ball of L{K, H). For A,XeB 
set 

Ma{X) = (1 - AA*)'^/^{A + X)(l + A*X)-\1 - A*Afl'^. (4.1) 

Clearly all are holomorphic on B. They are called the Mobius transformations. It can 
be proved that Mj^ = M_a (see [8], Theorem 2). Hence each Mobius transformation is 
a biholomorphic automorphism of B. Since Ma(0) = A the group of all biholomorphic 
automorphisms is transitive on B. 

We set 

p(A,5)=tanh-i(||M„^(S)||). (4.2) 

It is easy to see that p coincides with the Caratheodory distance in B. Indeed, by [6l 
Theorem 4.1.8], 0^(0, B) = tanh~^(||i?||) (this holds for the unit ball of every Banach space). 
Since cg is invariant and Ma sends A to we get: 

cb{A,B) =tanh~i ||M_^(5)|| =p{A,B). (4.3) 

Hence p is invariant with respect to biholomorphic automorphisms. I. Shafrir [15] proved 
that the space {B, p) is hyperbolic. We present a proof of this result in the appendix. 

A set in B is called bounded if it is contained in some p-ball, or equivalently in a multiple 
rB of the operator ball with r < 1. So a set is bounded if and only if it is separated from 
the boundary of B in the sense of Section [TJ 

The following lemma is a special case of a more general result proved in ^ Theorem 
IV.2.2]. 

Lemma 4.1. On any bounded set the hyperbolic metrics is equivalent to the operator norm. 
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5 WOT-topology 



As before, let B be the unit ball of the space of operators from K to H. We suppose that 
K is finite-dimensional, dimi^ = n, and that H is separable. We consider biholomorphic 
maps on B. By WOT we denote the weak operator topology (see [5\ p. 476]). Because of 
the separability, the restriction of this topology to B is metrizable, so in our arguments we 
may consider only sequences, not nets. 

Lemma 5.1. If K is finite- dimensional and H is separable then all biholomorphic maps of 
B are WOT -continuous. 

Proof. Let us firstly show that all Mobius transforms Mb are WOT-continuous (this was 
noticed and used already in the paper of Krein [H]). Indeed let B E B be fixed, then the 
map if : X ^ 1 + B*X from (;B, WOT) to {L{K, K),WOT). Moreover, since K is finite- 
dimensional, if remains continuous if instead of WOT we endow L{K, K) with its norm 
topology. The map T — > is norm continuous on the group of invertible operators on K. 
Hence the map ip : X ^ [1 + B*X)~^ is continuous from (jB, WOT) to L{K,K) with its 
norm topology. 

It follows that the map u : X ^ {X + B){1 + B*X)-^ is continuous from (S, WOT) to 
(i3, WOT). Indeed, if X^ ^ X, then u;(X„) - a;(X) = (X, + B){^{Xn) - V^(X)) + (X„ - 
X)ip{X), where i(j was defined above. The first summand tends to zero in norm while the 
second one tends to zero in WOT. 

By a result of Harris [7], if a biholomorphic map of B preserves the point 0, then it 
coincides with the restriction to B of an isometric linear map h : L{K, H) —>■ L{K, H). Since 
K is finite-dimensional, the WOT-topology on L{K, H) coincides with the weak topology 
(indeed L{K,H) is linearly homeomorphic to the direct sum of n copies of H); since any 
bounded linear map is weakly continuous, h is WOT-continuous. On the other hand, if is a 
biholomorphic map of B and A = ip{0) then ip = M_a o is a biholomorphic map preserving 
0. Hence ip is an isometric linear map and Lp = Mz\ o ip = Ma o is a composition of two 
WOT-continuous maps. Thus is WOT-continuous. □ 

Corollary 5.2. If dim K < oo and H is separable, then each ball E^^r 'is WOT -compact. 

Proof. Since there is a Mobius transform that maps E^^^ onto -Eo.r, and since all Mobius 
transforms are WOT-continuous, it suffices to consider the case A = But i?o,r is a usual 
closed operator ball; its WOT-compactness follows from the Banach-Alaoglu theorem. □ 

6 Restricted normal structure of B 

The purpose of this section is to show that in the case when dimi^ < oo and H is separable, 
the (open) operator ball B with the metric (14. 2 p has a restricted form of normal structure 
in the sense that WOT-compact p-convex subsets in it have non-diametral points. As we 
already mentioned B with the metric (14.21) is a hyperbolic space (see Section [TOj) . Our 
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assumptions on K and H imply that B is separable in the norm-topology and hence, by 
Lemma [4. H with respect to p. 

Theorem 6.1. Let K be finite dimensional and H he separable. Let M be a weakly compact, 
p-convex subset of B endowed with its hyperbolic metric. If M is not a singleton, then M 
contains a non-diametral point. 

Proof. Let a = diamM > 0. Assume the contrary, that is, all points in M are diametral. By 
Lemma ISTTl there is a sequence {An} in M such that lim„^oo p(^n, X) = a for each X G M. 

Since M is weakly compact, the sequence {An}'^=i contains a weakly convergent subse- 
quence. Let W be its limit, we have W E M (since M is weakly compact). 

Throughout this proof we will not change our notation after passing to a subsequence. 

Since W e M we get 

Mm p{W,Ar,) = a. (6.1) 

n— >oo 

We will get a contradiction by proving 

sup p{An, Am) > a. (6.2) 

n,m 

We may assume without loss of generality that W = Q (we can consider a Mobius 
transformation which maps W to 0, it is a p-isometry and weak homeomorphism) . 

Let /3 = tanha. Then ( 16. ip leads to lim„^oo ll^nll = and it suffices to show that 

sup||Ma,„(-AO|| >/3. 

71, m 

Since K is finite dimensional and An G L{K,H), we can select a strongly convergent 
subsequence in the sequence {y4*y4„}. Assume that A*^An — >■ P, where P G L{K,K). It is 
clear that P > and ||P|| = /J^. 

Choose e > and fix a number m with H^JI^Am — P|| < e. For brevity, denote A*^Am by 
Q. We prove that lim„_^oo ||^a„(— A„)|| > /? if e > is small enough. By the definition, 

MA,A-An) = (1 - AmAl,r^'\Am - An){l - Al,Anr\l - Al,Amf'\ (6.3) 

Since A*^ is of finite rank A^An — ^ in the norm topology. Hence lim„^oo ||^Am(— ^n)|| = 
lim^^oo \\Tn\\ where 

Tn — 0- ~ AmA^) ^ {Am — An){l — A^Am) ^ = Am — (1 — AmA^) ^ An{l — A^^Am) ^ ■ 

It follows from the identity 

^ ^ (l-t)(l + (l-t)-V2) 



7 



that the operator (1 — A^A^) is a finite rank perturbation of the identity operator. Since 
A„ ^ in WOT, we obtain that ||r„ - S'„|| 0, where Sn = - - Al^A^y/'^. 
Denote - A*^Amy/'^ by 5„. Since 5„ ^ in WOT, the sequence 

{Ara — Bn) {Am — Bn) — A^Am — B^Bn = —A^Bn — B^Am 

tends to zero in norm topology. Furthermore, 

BlB^ = {l-QY'^M^A^{l~QY'^ 
tends in norm topology to (1 — Q)^/^P(1 — QY^"^- Therefore 

{Ar^ - B^r{Am _ 5j ^ g + (1 _ g)V2p(i _ g)i/2_ 

Since ||P — QH < we have that 

\\Q + (1 - QY''P{i - QY^' - (Q + (1 - Q)Q)\\ < e. 

The inequalities 

P'-s<\\Q\\<p' 

imply 

\\Q + {l-Q)Q\\>2p^-P'-2e, 

whence 

hm ||5:^„||= lim - i?„)*(A„ - II > 2/32-/3^-35 >/32, 

if e is sufficiently small. □ 

7 Fixed points 

The main purpose of this section is to establish the existence of a common fixed point for 
an elliptic group G of biholomorphic transformations of the operator ball B. As is shown in 
Section [in] a biholomorphic transformation of ;B is a bijective isometric transformation of the 
metric space p) which maps the set M. onto itself (and hence segments onto segments). 

Lemma 7.1. If G is an elliptic group of biholomorphic transformations ofB, then there is 
a non-empty WOT -compact p-convex G-invariant subset of B. 

Proof. Let A E B he such that the orbit G{A) := {g{A) : g G G} is bounded. Therefore 
G{A) is contained in some closed ball £'a,r- Let M be the intersection of all closed balls 
containing G{A). It is clear that this intersection is non-empty (it contains G{A)), WOT- 
compact and p-convex (as an intersection of WOT-compact p-convex sets). It remains to 
check that it is G-invariant. To see this it suffices to observe that each element g E G maps 
the set of balls containing G{A) bijectively onto itself. □ 
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Lemma 7.2. Let G be an elliptic group of biholomorphic transformations of B. Let M be a 
minimal WOT -compact p-convex G-invariant subset in {B,p). Then M is a singleton. 

Proof. We use the approach suggested in [4j. Assume the contrary, let diamM = a > 0. 
By Theorem 16.11 M contains a non-diametral point A^, so that M C {A : p{A, N) < 6} for 
some 5 < a. Consider the set 

O = fl Eb,s. 

BeM 

The set O is non-empty because N E O. The set O is weakly compact and p-convex since 
each of the balls Eb^s is weakly compact and p-convex. The set O is a proper subset of M 
since M has diameter a > 6. 

Since G is a group of isometric transformations and M is invariant under each element 
of G, the action of G on M is by isometric bijections. Therefore O is G-invariant. We get a 
contradiction with the minimality of M. □ 

Proof of Theorem 11.11 The implication (i) =^ (ii) is obvious. On the other hand if 
G{Xq) is separated from the boundary, for some Xq E B then supg^Q p{0, g{Xo)) < oo 
whence, for each X e B, snpg^a p{0, g{X)) < snpg^a{p{0, g{Xo)) + p{g{Xo), g{X))) = 
supgg(^(p(0, giXo)) + p(Xo, X)) < oo. This means that the orbit G{X) is separated from the 
boundary. We proved that (i) (ii). 

The implication (i) =^ (iv) can be derived from Lemmas 17. II and 17.21 as follows. It is clear 
that families of WOT-compact p-convex G-invariant sets with the finite intersection prop- 
erty have non-empty intersections which are also WOT-compact p-convex and G-invariant. 
Therefore, by the Zorn Lemma, there is a minimal non-empty WOT-compact p-convex G- 
invariant set Mq. By Lemma [7.21 Mq is a singleton and (iv) is proved. 

If (iv) is true and A is a fixed point of G then, Gi = M_aGMa is a group of biholomorphic 
maps of B preserving 0. Hence it consists of restrictions to B of isometric linear maps (see 
the beginning of Section H] in this connection). Thus Gi is equicontinuous. 

Note that each Mobius transform is a Lipschitz map: ||M^(X) — M^(y)|| < C\\X — Y\\ 
for each X,Y G B, where the constant G > depends on A. Indeed setting F{X) = 
{A + X){1 + A*X)'^ and D = {1 - \\A\\)-^ we have 

\\F{X) - F{Y)\\ = \\{A + X)((l + A*X)-^ - (1 + A*Y)-^) + {X - Y){1 + A*Yy^\\ 

= \\{A + X){1 + A*X)-'^A*{Y - X){1 + A*Yy^ + {X - Y){1 + A*Y)-^\\ 
< 2D^\\X - Y\\ + D\\X - Y\\ < 3D^\\X - Y\\. 

Hence 

||Ma(X) - Ma{Y)\\ = 11(1 - AA*)-^/^{F{X) - F{Y)){1 - A*Ay/^\\ 
< D^\\F{X) - F{Y)\\ < 3dI\\X-Y\\. 

Since G = MaGiM^a and the maps Ma, M^a are Lipschitz, G is also equicontinuous. 
We proved that (iv) =^ (iii). 
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Let now (iii) hold, we have to prove (ii). We will show that the orbit of is separated 
from the boundary. Assuming the contrary we get that for any 6 > there is g E G with 
||5f(0)|| > 1- 6. Let A = g{0); we may assume that 5 < 1/2 so \\A\\ > 1/2. 

By the already mentioned result of |7], 5^ = Ma o h where /i is a linear isometry. Let P 
be the spectral projection of T = A* A corresponding to the eigenvalue ||T|| = ||v4|p (recall 
that T is an operator in a finite dimensional space). Then 



'l-T)P\\ = l 



Set Xi = 0, X2 



On the other hand 



h'^(^AP). Then \\X2 



- ||T| 
^ill 



< 2(1 



\A\ 



) < 26. 

1^11/2 > 1/4. 



-AP 

A A*] 



'1/2 



A{l-T) 



Al-P 



-1/2 



Up 
2 



2^ + ^ 



1 + -TP 



A 



-A*AP 



'1-A*A)^/^ - A 



1 + -TP 



A 



[l-T) 



1/2 



A 



-A{l-T)P( 1 + -TP 



<-\\A\\\\{l-T)P\\<-26 = 6. 

This contradicts to the assumption of equicontinuity. Indeed for each 6 we get points 
Yi = g{Xi) with ||Yi - F2II < S and \\g-\Yi) - g-\Y2)\\ > 1/4. Thus (ii) holds. □ 



8 Orthogonalization 

Theorem 8.1. If a bounded representation tt of a group G on a Hilbert space Ti preserves 
a quadratic form t] with finite number of negative squares then it is similar to a unitary 
representation. 

Proof By our assumptions, Ti = Hi (BH2, dim(if2) < 00, and ri{x) = ||-Pa;|p — HQxp where 
P, Q are the projections onto Hi and H2 respectively. We write Hi = H and H2 = K, for 
brevity. 

We will relate to each invertible operator T on ?i preserving the form 77 a biholomorphic 
map wt of B in such a way that 

WT1T2 = U!Ti O UJT2- (8-1) 

Let us call a subspace L of H positive (negative) if ri{y) > (respectively ri{y) < 0) for 
all non-zero y E L. Since each negative subspace L is finite-dimensional, there is £ > such 
that 

f]{y) ^ s-ll non-zero y E L. 
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The supremum of all such e is called the degree of negativeness of L and is denoted by e(L). 



For each operator A & B, the set 



L{A) = {Ax®x -.xeK} 



is a negative subspace of 7i. Furthermore the condition 



v{y) ^ "^llz/ll^) for y ^ -^(^) 



means that 



xf + \\Axf < -e{\\x\\^ + pxf ) 



for all X G -ft'. That is 



e < 



1 - pf 
1 + IMP' 



It follows that the degree of negativeness of L{A) is related to \\A\\ by the equality 



Since dim(L(y4)) = dim(i^), L{A) is a maximal negative subspace in Ti. Indeed if some 
subspace M oiTi strictly contains L{A) then its dimension is greater than codimension of 
whence M P\H ^ {0}. But all non-zero vectors in H are positive. 

Conversely, each maximal negative subspace Q oiTi coincides with L{A), for some A & B. 
Indeed, since Q HH = {0}, there is an operator A: K ^ H such that each vector of Q is of 
the form Ax®x. Since Q is negative, we have ri{Ax®x) = — < 0, and therefore 

< 1, so A e B. Thus Q C L{A); and, by maximality, Q = L{A). 

It is easy to see that the map A L{A) from B to the set S of all maximal negative 
subspaces is injective and therefore bijective. 

Now we can define wt- Indeed, if a subspace L of ?i is maximal negative then its image 
TL under T is also maximal negative (because T is invertible and preserves rj). Hence, for 
each Ae B, there is B e B such that L{B) = TL{A). We let wriA) = B. 

The equality (18.11) follows easily because L{wtj^{wt2{A))) = TiL{wt2{A)) = TiT2L{A) = 
L{wtiT2{^)) S'lid the map A — >■ L{A) is injective. 

Our next goal is to check that wt is biholomorphic. Since w^^ = wt-^ it suffices to show 
that Wt is holomorphic. 

Let T = (Tij)f j^-^ be the matrix of T with respect to the decomposition Ti = Hi ® H2- 
Then T{Ax © x) = [TuAx + Tisx) © (TaiAx + T22X). Since T{Ax © x) G L(wt{A)), we 
conclude that 



e{L{A)) 



l-\\Af 
1+ MP' 



(8.2) 



wt{A){T2iAx + T22X) = TuAx + T12X. 



Thus 



wt{A) = {TuA + Ti2){T2iA + T22) 



-1 



(8.3) 
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This shows that wt is a holomorphic map on B. 

Suppose now that vr is a bounded representation of a group G on TC preserving rj. Then 
W = {w-jrig) '■ 9 £ G} is a group of biholomorphic maps of B. Moreover since vr is bounded, 
the group W is elhptic. To see this, note that for each negative subspace L, one has 

7]{y) < -e{L)\\yf ioi all y G L. 

If T is an invertible operator preserving r] then T~^x G L, for each x G TL, whence 

r^{x) = r]{T-^x) < -e{L)\\T^^xf < -e{L)\\T\\-'^\\x\\'^. 

Thus 

e{TL)>eiL)\\T\\-'. 
For L = L{A),TL = This gives 

1 + \\wt{AW - " " l + JW 
if one takes into account (18. 2p . Thus, if ||vr(5f)|| < C for all g E G, then 



Therefore 



and 



1 + 1 + PII 

2 ^ ^^-2^ - 11^1 



\wM,){A)r>c 



sup < 1 

<?6G 

for each A E B. 

By Theorem 11.11 there is D E B with WTr[g){D) = D for all g E G. Hence T[{g)L{D) = 
L{D) for all ^ G G. 

Let U be an operator on H with the matrix (Uij) where Uu = {1h — DD*)^^^"^, Uu = 
-D{1k - D*D)-^/^, U21 = -D*{Ih - DD*)-^/^, U22 = (Ik - D*D)-^/^. It can be checked 
that U preserves 77 and maps L{D) onto K. Then all operators T{g) = UTx{g)U~^ preserve 
ry, and the subspace K is invariant for them. It follows that H is also invariant for operators 
T{g). Hence these operators preserve the scalar product on Ti. Thus g i— > T{g) is a unitary 
representation similar to vr. □ 

It should be noted that Theorem 18.11 does not extend to the case when rj has infinite 
number of negative (and positive) squares, that is, to the case that both Hi and H2 are 
infinite-dimensional [T6l. 
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9 J-unitary operators on Pontryagin spaces 



The Pontryagin space is a linear space Ti supplied with an indefinite scalar product x,y ^ 
[x, y] which has a finite number of negative squares. More precisely this means that one 
can choose a usual scalar product x, ?/ — > {x, y) with respect to which £^ is a Hilbert space 
and [x,y] = {Jx,y), where J is a selfadjoint involutive operator on this Hilbert space with 
rank(l — J) < oo. An invertible operator T on £^ is called J-unitary if [Tx,Ty] = [x,y] for 
all x,y E S. 

It should be noted that the terminology does not seem to be successful because the 
choice of the operator J and the corresponding scalar product is not unique while the set of 
J-unitary operator is completely determined by the original indefinite scalar product [-,■]. 
However, this terminology is widely used (see, for example, pQ, [TD] and references therein). 
It is important that all scalar products defining [-, ■] via J-operators are equivalent, so one 
can speak, for example, about boundedness of a set of operators, without indicating which 
scalar product is chosen. 

A subspace X C S is called positive (negative) if [x,x] > (respectively [x,x] < 0) for 
all X E X. A dual pair of subspaces in £^ is a pair Y, Z, where F is a positive subspace, Z 
is a negative subspace and Y + Z = S. The study of dual pairs invariant for a given set of 
J-unitary operators was started by Sobolev and intensively developed by Pontryagin, Krein, 
Phillips, Naimark and other prominent mathematicians. 

The previous theorem on the orthogonalization of representations implies the following 
result. 

Corollary 9.1. A group of J-unitary operators on a Pontryagin space has an invariant dual 
pair if and only if it is bounded. 

Proof. Choose a scalar product (-, ■) and the corresponding operator J. Denote by H the 
Hilbert space [S, {■,■)). Since J is an Hermitian involutive operator, there are orthogonal 
subspaces H, K of H such that J = Ph — Pk- By our assumption on J, the subspace K is 
finite-dimensional. 

Let G be a group of J-unitary operators. If it is bounded, then the identity map can be 
regarded as a bounded representation of G on Ti. Moreover it preserves the form ri{x) = 
[x,x]. Since it has a finite number of negative squares. Theorem 18.11 implies that there is an 
invertible operator V such that the representation T{g) = T~^gT is unitary. It follows from 
Theorem 5.8] that G has an invariant dual pair of subspaces. 

For completeness we include the proof of this fact. Passing to adjoints in the equality 
Trig) = gT and taking into account that g* = Jg~^J, T{g)* = T{g~^) we obtain that 
T{g^^)T* = T*Jg^^J. Using this identity for g instead of g~^ and multiplying both sides by 
JT we get: 

T{g)T*JT = T*JgJJT = T*JgT = T*JTT{g). 

Thus the invertible selfadjoint operator R = T*JT commutes with the group t{G) of 
unitary operators. It follows that its spectral subspaces Hi and Ki corresponding to positive 
and negative parts of spectrum are invariant for t{G). Note that {Rx,x) > for x G 
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T-^H\{0} and {Rx,x) < for x G T-^K\{0}. It follows that dimiTi = dimK. Now the 
subspaces H2 = THi and K2 = TKi form an invariant dual pair for G. 

The converse implication is simple. If G has an invariant dual pair if, K then the scalar 
product {hi + ki,h2 + /C2) = [hi, h2\ — [fci, ^2] is invariant for G. Thus G is a group of unitary 
operators on 7i = (-, ■)), hence it is bounded. □ 

As a consequence we obtain the following result proved in [T6] : 

Corollary 9.2. A J -symmetric representation of a unital C* -algebra on a Pontryagin space 
is similar to a * -representation. 

For a proof it suffices to notice that restricting the representation to the unitary group 
of the C*-algebra we obtain a bounded group of J-unitary operators. 



10 Appendix: Hyperbolicity of B (after Itai Shafrir) 

For any bounded domains Di,D2 of complex Banach spaces we denote by Hol{Di, D2) the 
set of all holomorphic maps from Di to D2. U Di = D2 = D then Hol{Di, D2) is a semigroup 
with respect to the composition, and by Aut{D) we denote the set of all its invertible elements 
(biholomorphic automorphisms of D). The group Aut{B) acts transitively on B. Indeed, for 
each A E B the Mobius transform M4 is biholomorphic and sends to A 
As usually the Caratheodory metric on B is defined by the equality: 

csiA, B) = sup{^(/(A), fiB)) : / G HoliB, A)} 

where A is the unit disk and uj is the Poincare distance: 

u{zi, Z2) = ta.nh~^ ^ 



- Z1Z2 

As it was mentioned in Section 4, cq coincides with the metric p defined by the formula 
(14. 2p . Clearly cg is invariant under biholomorphic maps of B. 

We shall prove that ;B is a hyperbolic space with respect to this metric. 

Furthermore the differential Caratheodory metrics on B is defined by 

«(AV^)= sup f^\f)'!\2 (10-1) 

feHol(B,A) 1 - 1/(^)1 

for all A E B,V & L{K,H), where T>f{A) is the differential of / in A (see |j6j, where a is 
denoted by 7/3). 

Lemma 10.1. For each A e B, V e L{K, H) 

VMb{A)V = (1 - BB*Y''^{1 + AB*)-W{1 + B*A)-^{1 - B*BY''^. (10.2) 
In particular, 



VMb{0)V = (1 - BB*y^^V{l - B*B) 



1/2 
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Proof. By definition, Mb(X) = {1 - BB*)-^/^{B + X){1 + B*X)-\l - B* By/'^. We have to 
calculate the coefficient c of t in the Taylor decomposition of the function t Mb{A + tV). 
For this, note that if P is an invertible operator then (P + tQ)~^ = P~^ — tP^^QP^^ + o(t). 
It follows immediately that 

c = (1 - BB*y^/\V{l + B*A)-^ -{B + A){1 + B*A)-^B*V{1 + B*A)-^){1 - B^Bf^ 
= (1 - BB*)-^'^{1 -{B + A){1 + B*A)-^B*)V{1 + B*A)-^{1 - B*Bf'^ 
= (1 - BB*)-^'^{1 -{B + A)B*{1 + AB*)-^)V{1 + B*A)-^{1 - B*BY^^ 
= (1 - BB*)-^'^{{1 + AB* -{B + A)B*){1 + AB*)-^)V{1 + B*A)-\l - 5*5)^/2 
= (1 - BB*y/\l + AB*)-^V{1 + B*A)-\1 - B*BY'^. 

□ 

Lemma 10.2. a{A,V) = - AA*)-^/^V{l - A*A)-^/'^\\ for all A e B andV e L{K,H). 
Proof. By [Gj Lemma V.1.5] 



a{0,V) = 

Let now A be arbitrary. Then by [6l Proposition V.1.2] 

a{MA{0),VMA{0)X) = a{0,X) = \ \X\\. 

On the other hand, by Lemma IIO.H 

a(M^(0), PM^(O)X) = a{A, (1 - AA*y/^X{l - A*A f^). 

Setting now V = {1 - AA*y/^X{l - A*Ay/\ we obtain X = (1 - AA*)-^/'^V{l - A*A)-^/^ 
and hence a{A,V) = \ - AA*)-^/^V{l ~ A* Ay^/^)\\. □ 

For any bounded operator D, set D'-^^ = D, D^^^ = DD*D, D^^^ = DD*DD*D, 
^(2fc+i) _ {^DD* fD. 

Let 

oo 

ThD = ^a2„+iD(''^+^) (10.3) 

n=0 

where are the Taylor coefficients of tanht, i.e., tanht = 'Yl^=Q^'2n+it'^"'^^ ■ 
It follows from the definition that Th D = tanh(D) if D is selfadjoint. 
If D = J\D\ is the polar decomposition of D (that is, \D\ = (D*Dy/'^ and J is a partial 

isometry such that {J*J)\D\ = \D\{J*J) = \D\), then 

2^(2n+l) ^ J|/)|2n+l^ 

and hence 

Th D = JtanhlDl. 



15 



On the other hand, we can write D = \D*\J, where \D*\ = J\D\J* = {DD*y/^, therefore 

ThD = (tanh|D*|)J. 

For the space {B,p) we define the set Ai of fines as foUows: for A E B, D E dB (i.e., 
||L>|| = 1) we let 

1A,D = ilAAt) ■■= M^(Tfi(tD)) : t G M} (10.4) 

and set 

M = {ia,d: AeB.D edB}. 
Proposition 10.3. 7a,_d ^-5 ^ metric line. 

Proof. It is enougfi to sfiow tfiat p(7a,d('S), 7yi,D(^)) = 1-5— Since p is invariant witfi respect 
to Ma we can assume tfiat A = 0. We fiave p(7o,d(s), 7o,z)(i)) = tanfi""*^ ||MB(Tfi(t_D))||, 
wfiere B = — Tfi(sD). Using polar decomposition D = J\D\ we fiave tfiat Th.{tD) = 
Jtanfi(i|D|), Tfi {WyTh (sD) = tanfi(t|L'|) tanfi(s|L'|), wfience 

MniTHtD)) = {l-Th{sD)Th{sDy)-^/^{T\i{tD)-Th{sD)) 

(1 - Th{sDyTh{sD))-\l - Tfi(sD)*Tfi(sD))i/2 

= J(l - tanfi2(s|D|))"^/VV(tanfi(t|D|) - tanfi(s|D|)) 
(1 -tanfi(s|D|)tanfi(t|Z}|))-i(l -tanfi2(s|D||))^/2 

= Jtanfi((t-s)|D|) = Tfi((t-s)D) 

giving tfie statement. □ 

We fiave to prove tfiat '~fA,D{t) is a metric curve, in tfie sense tfiat tfie metric of its 
derivation equals 1, i.e., Q;(7(t), 7'(t)) = 1. 

Lemma 10.4. Let 7(t) = Th{tD), D G dB. Then 

i{t)=D-^{t)D*^{t). (10.5) 
Proof. We fiave 7(t) = Jtanfi(t|L'|) and 

7'(t) = J\D\ cosfi(t|D|)"2 = D(cosfi(t|D|)~l 

On tfie otfier fiand 

D--f{t)D*^{t) = D-Jtanh{t\D\)\D\J*Jtanh{t\D\) = D-Dta.nh\t\D\) = D{cosh{t\D\))-^ 
giving (11031) . □ 
Lemma 10.5. Let 7(t) = Th{tD), D G dB. Then 

(1 - 77*)-^/2p _ ^D*-f){l - 7*7)-'/' = D. (10.6) 
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Proof. Setting D = J\D\, we have 

7*7 = Th{tDyTh{tD) = t&nh{t\D\)J* J t&nh{t\D\) = tanh2(t|D|). 

Furthermore 

77* = tanh(t|D*|)JJ*tanh(t|D*|) = tanh\t\D*\). 

Since 

D\D\ = \D*\D, 

we have that 

Dfi\D\) = fi\D*\)D 
for any bounded Borel function /. Taking /(x) = 1 — tanh^(ta;), we get 

Z}(1-7*7) = (1-77*)D. 

Next 

D*-f = 7*L) 

because D*Th{tD) = D* J tanh{t\D\) = \D\tanh{t\D\) is selfadjoint. 
Hence 

(1 - 77*)-i/2p _ 7D*7)(1 - 7*7)-'/' = (1 - 77*)"'/'(l - 11*)D{1 - 7*7)-'/' 

= (1 - 7*7)'/'(l - 7*7)-'/2^ = D. 

□ 

Proposition 10.6. Let ^{t) = 7A,D(t), A E B, D e dB. Then 

a(7(t),7'(t)) = l. 

Proof. It suffices to prove this for A = 0, since a{F{X),VF{X)V) = a{X,V) for any 
F G Aut{B), X eB,V e L{H,K) (see [6, Proposition V.1.2]), and hence 

a{MA{THtD)),{MA{THtD))y) = a(M^(Th(tD)), DMA(Th(tZ}))(Th(tZ}))') 

= a(Th(tD),(Th(tD))'). 

Assume therefore 7(t) = Th(tD). By Lemma [1U.2I and I1U.5I we have 

«(7W,7'W) = ll(l-77*)-'/'p-7^*7)(l-7*7)-'/'ll = 11^11=1. 

□ 

The next step is to prove that the family A4 of all lines is invariant with respect to the 
biholomorphic maps of B. 

Lemma 10.7. Let T]{t) = MA{'y{t)) where 7(t) = Th(tD). Then, for each biholomorphic 
map h : B ^ B, the curve h{ri{t)) belongs to the family Ai. 
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Proof. By [HI Theorems 3 and 4], there is a hnear isometry L of the space L{K, H) to itself 
satisfying the condition 

L{AB*A) = L{A)L{ByL{A) for all A,B e L{K, H) (10.7) 

and such that 

h = Mft(o) oL = Lo M_h(Qy 
It follows from fll0.7l) (see a remark after [8^ Corollary 5]) that 

LoMa = Ml{a) o L 

for all AeB. 

So it suffices to consider the cases h = L and h = Mb- Let us firstly prove that 
L{r]{t)) eM. Indeed, 

L{rj{t)) = L{MA{l{t)) = Mi(^)(L(7(t)) 

= MLiA){L{Th{tD))) = M^A){THtL{D))) e M. 

Now we have to prove that MB{f]{t)) G M.. Applying [U Theorems 3 and 4] to h{x) = 
Mb{Ma{x)) we get a linear isometry L satisfying (110.71) and such that 

Mb o Ma = Mc o L 

where C = h{0) = Mb(A). Thus 

MBivit)) = MB{MA{l{t)) = Mc{L{^m = Mc{Th{tL{D))) e M. 

□ 

Our next goal is to show that for each A,B & B there is a unique line in A4 which passes 
through A, B. 

Lemma 10.8. The set of all lines in M. that go through A is {'^a,d '■ D G d{B)}. 

Proof It suffices to assume that A = 0. Suppose that a line 7(t) = MB{Th.(tD)) goes 
through 0, i.e., 7(s) = for some s G M. Then clearly B = — Th(sD). Using the arguments 
from the proof of Proposition 110.31 we obtain 7(t) = Th((t — s)D). Thus 7 = 7o,_d- D 

Corollary 10.9. For each A,B & B, there is a unique line in A4 that passes through them. 

Proof. We may assume that A = 0. Let B = J\B\ be the polar decomposition of B and let 
C = tanh"^ 1-^1/^0 for to > be such that ||C|| = 1. Then for D = JC the line 7o,d passes 
through and B. 

If there are two lines, 7o,i?i and 70,^2 ; going through B then by the above lemma, B = 
Th.(tDi) = Th{sD2) for some t, s G M. We may suppose that t, s > 0. Taking polar 
decompositions of Di = Ji\Di\ and D2 = J2I-D2I we see that Ji = J2 and tanh(i(:|Z}i|) = 
tanh(s|D2|)! which imply that t|-Di| = s|D2|. But this clearly shows that the lines coincide. 

□ 
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Lemma 10.10. 

\\A\\ < 11(1 - BB*)-^/\A - BA*B){1 - B*B)-^/^\\. (10.8) 

for each A, B e B. 

Proof. Consider the polar decomposition B = J\B\. Then = {BB*Y^'^ = J\B\J*. Let 
P = tanh"^(|fi*|), and Q = tanh-^(|5|). Then 

(1 - BB*)-^/\A - BA*B){1 - B*B)-^'^ = {coshP)A{coshQ) - (sinhP) JAJ*(sinhQ). 

For any e > 0, there are unit vectors x, y such that 

((coshP)A(coshQ)x,?/) > ||(coshP)?/||||A||||(coshg)a;|| - e. 

Since || (cosh P)?/p — ||(sinhP)y||^ = and || (cosh (5)a;||^ — || (sinh = one can 
find numbers a, b such that 

II (sinh P)|/|| = sinh6, ||(coshP)?/|| = cosh 6, || (sinh (5)a;|| = sinha, ||(cosh(5)a;|| = cosh a. 

Hence 

||(coshP)A(coshg) - (sinhP)JAJ*(sinhQ)|| 

> (((coshP)A(coshQ) - (sinh P)JA J* (sinhQ))a;,?/) 

> (cosh6)(cosha)||yl|| —e — (sinh6)(sinha)||A|| 

> cosh(6-a)p|| - £ > pll -e, 

giving the statement. □ 
Lemma 10.11. Let us consider two lines: 7(t) = M^(Th(tC)), r]{t) = MA(Th(tP))). Then 

2p{j{s),r]{s))<p{j{2s),r]{2s)) (10.9) 

for each s > 0. 

Proof. Since p is invariant with respect to the transformations Ma we may assume A = 0. 

Let C{t) be a curve '~iB,E{t) which joins 7(2s) with rj{2s), we assume that C(0) = 7(2s), 
Cito) = r){2s) for some to > (such curve exists by Corollary 110. 9p . Define now a new curve 
Cihy 

Ci = Th ( -Th-^C 



Then Ci(0) = j{s), Ci{to) = ri{s) and 

C{t) = 2Ci(t)(l + Ciit)*Ciit))-\ (10.10) 



As usually we denote by L{Ci) the length of the curve Ci. L{Ci) = /q° Q;(Ci(t), C[{t))dt. 
If we could show that 

L{C) > 2L{Ci) (10.11) 
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for all curves C, Ci satisfying (110.101) then we would obtain that 

p(7(2s),r7(2s)) = L{C) > 2L{C,) > 2p{^{s)Ms)) 

(the first equality follows from Proposition 110.31 and Proposition 110.61 the last inequality 
holds because the length of any curve is not smaller then the distance between its ends). 
Thus our goal is the inequality (110.111) . It suffices to show that 



Since 
we have 
whence 



2a(Ci(t), C[{t)) < a(C(t), C\t)). (10.12) 

2Ci = C(l + C*Ci), 
C\l + ClCi) + C{C'*Ci + CIC[) = 2C[, 

C' = ((2 - CCl)C[ - CC[*Ci){l + ClCi)-\ (10.13) 



Since 

2 - CC* = 2 - 2Ci(l + C*Ci)-^C* = 2(1 - CiC*(l + CiC*)"^) = 2(1 + CiC*)-\ 

substituting this into (110. 13p we obtain 

C = 2((1 + CiCl)-^C[ - Ci{l + ClCi)-^C'*Ci){l + ClCi)-^ 

= 2(1 + CiCl)-\C[ - CiC'*Ci)il + ClCi)-\ 
Now it follows from Lemma 110.21 that the inequality (110.111) is equivalent to the following 

11(1 - CiC*)-^/^C[{l - c*Ci)-^/^\\ 



< 11(1 - cc*)-^/\i + CiCi)-\c[ - CiC'*Ci)(i + c;Ci)-\i - c*c)-^^^\\. 



But 



1 - CC* = 1 - 4Ci(l + ClCi)-^Cl = 1 - 4CiC*(l + CiCl)-^ 
= ((1 + CiClf - 4CiC*)(l + CiCl)-^ = (1 - CiCl)\l + CiCl)-^. 

Similarly 

(1 - C*C)-i/2 _ (1 ^ C*Ci)(l - C*Ci)-\ 
It follows now that (110. 14p is equivalent to the inequality 

- CiCl)-^/^C[{l - ClCi)-^/^\\ < - CiCl)-\C[ - CiC'*Ci){l - ClCi)-^\\. (10.15) 

But (fTaT5|) follows from Lemma[IOlO]by substituting B = Ci and A = {l-CiCl)-^/'^C[{l- 
C*Ci)-V2 into inequality ffTaSj) . □ 
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The above results establish 



Theorem 10.12. B is a hyperbolic space. 
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